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SUMMARY 


Problem 


Selection  and  classification  of  recruits  is  currently  done  on  the  basis  of  the  cognitive 
abilities  measured  by  the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB  t  There  is 
a  need  for  a  more  broadly  based  system  that  considers  a  range  of  variables  in  addition  to 
general  cognitive  ability.  A  first  step  toward  achieving  this  end  is  to  develop  a  taxonomy 
that  classifies  Navy  ratings  on  the  basis  of  the  variables  that  are  most  likely  to  predict 
job  performance.  Such  a  taxonomy  would  be  particularly  useful  for  interpreting  research 
on  the  ASVAB,  as  well  as  research  on  the  validity  and  utility  of  the  experimental  tests 
that  are  currently  being  developed  by  the  Navy  and  other  services. 

Purpose 

The  purpose  of  this  research  is  to  develop  and  apply  a  method  of  classifying  Navy 
ratings  on  the  basis  of  the  variables  that  predict  successful  performance  on  the  job.  The 
research  described  here  comprised  five  separate  activities:  (1)  determining  appropriate 
dimensions  for  classifying  the  set  of  relevant  and  practiced  predictors  (job  requirement 
dimensions),  (2)  selecting  a  representative  sample  of  ratings  for  study,  (3)  collecting  job 
requirement  data  using  multiple  methods,  (4)  assessing  the  convergence  between  data 
collected  using  different  methods,  and  (5)  classifying  ratings. 

Approach 

Five  job  requirement  dimensions  were  defined  that  showed  sufficient  breadth  and 
relevance  to  serve  as  a  basis  for  classifying  Navy  ratings.  These  dimensions  were:  (1) 
cognitive  ability,  (2)  psychomotor  ability,  (3)  information-processing  ability,  (4)  trainings 
nd  (5)  experience. 

Four  methods  were  then  used  to  assess  the  usefulness  of  each  of  these  dimensions  in 
predicting  job  performance:  (1)  Occupational  Ability  Pattern  analysis,  in  which  informa¬ 
tion  is  obtained  from  the  Dictionary  of  Occupational  Titles  regarding  ability  and  training 
requirements  for  jobs;  (2)  Expert  Judgment,  in  which  Chiefs  (E-7,  E-8,  E-9)  in  the  ratings 
studied  served  as  subject-matter  experts;  (3)  Validation  Research,  which  involves  ac¬ 
cumulating  results  over  several  validity  studies  to  estimate  the  usefulness  of  different 
predictors  in  groups  of  ratings,  and  (4)  Task  Analysis  of  each  of  the  ratings  studied.  The 
convergence  between  these  four  methods  was  assessed,  and  a  rating  taxonomy  was 
developed  on  the  basis  of  data  from  the  measurement  methods  that  showed  the  most 
convergence. 

Results 


Data  obtained  from  Expert  Judgment,  Validation  Research,  and  Task  Analysis  showed 
significant  agreement  regarding  the  usefulness  of  the  five  job  requirement  dimensions  for 
predicting  performance  in  a  sample  of  18  ratings  that  are  broadly  representative  of  Navy 
activities;  data  from  Occupational  Ability  Pattern  analysis  showed  less  agreement.  On 
the  basis  of  Expert  Judgment  data,  the  ratings  were  classified  into  three  groups;  (1 ) 
cognitive  group,  (2)  practical  ability  group,  and  (3)  training  group.  In  the  cognitive  group, 
general  cognitive  ability  was  seen  as  the  major  determinant  of  job  success.  The  practical 
ability  group  emphasized  perceptual-motor,  problem-solving, a  nd  information-processing 
skills.  Finally,  the  training  group  emphasized  performance  in  training,  and  de-emphasized 
cognitive  and  perceptual  abilities  as  determinants  of  job  success. 
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Conclusions 


Expert  judgment  provides  a  practical  and  valid  method  of  obtaining  job  requirement 
data.  These  data,  in  turn,  can  be  used  to  form  a  predictor-based  taxonomy  of  Navy 
ratings.  This  taxonomy  suggest  that  some  tests,  particularly  those  that  measure 
perceptual-motor,  problem-solving  and  information-processing  skills,  will  be  significantly 
more  useful  in  some  ratings  than  in  others.  Thus,  the  results  of  validity  studies  for  this 
class  of  test  could  depend  on  the  sample  of  ratings  included  in  the  study. 

Recommendations 


The  method  developed  here  provides  a  practical  basis  for  grouping  Navy  ratings,  and 
the  resulting  taxonomy  appears  to  have  implications  for  the  design  and  evaluation  of 
selection  and  placement  systems.  The  first  recommendation  is  that  the  methods 
developed  here  be  applied  to  the  remaining  ratings  in  the  Navy.  This  will  allow 
assessment  of  the  validity  and  generalizability  of  the  taxonomy  that  has  been  developed  in 
the  current  sample  of  ratings.  Second,  the  taxonomy  leads  to  different  tests  in  different 
groups  of  ratings.  As  job  performance  measures  are  developed  in  these  ratings,  the 
validity  of  these  predictions  should  be  assessed.  Additional,  more  specific  recommenda¬ 
tions  are  presented. 
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INTRODUCTION 


Problem 


Many  of  the  major  personnel  functions,  including  personnel  selection,  placement,  and 
training,  depend  in  part  on  a  knowledge  of  the  variables  that  are  related  to  successful 
performance.  There  is  extensive  evidence  that  individual  differences  in  cognitive  ability 
are  related  to  job  performance.  There  is  also  evidence  that  abilities  outside  of  the 
traditional  cognitive  domain,  including  psychomotor  abilities  and  information-processing 
abilities,  predict  job  performance  (Fleishman  &  Quaintance,  1984).  Although  general 
intelligence  is  thought  to  be  related  to  performance  in  practically  all  jobs  (Schmidt  & 
Hunter,  1981),  it  is  not  clear  whether  all  cognitive  and  noncognitive  abilities  are  equally 
relevant  for  predicting  performance  on  different  jobs.  The  use  of  ability  tests,  such  as 
the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB),  in  classification  is  based  on  the 
assumption  that  different  abilities,  aptitudes,  and  skills  are  needed  in  different  jobs 
(Cronbach  &  Snow,  1977). 

Although  it  is  likely  that  different  jobs  do  require  different  abilities  and  skills,  it  is 
unlikely  that  each  job  is  totally  different  in  its  requirement  from  every  other  job.  It  is 
more  likely  that  jobs  can  be  grouped  on  the  basis  of  their  ability  and  skill  requirements 
(Pearlman,  1980).  Such  a  taxonomy  would  be  useful  for  several  purposes,  especially  in  the 
design  and  validation  of  personnel  selection  and  placement  systems.  For  example, 
considerable  attention  has  recently  been  given  to  the  development  of  computerized  tests 
of  spatial  and  perceptual  abilities.  It  is  unlikely  that  these  tests  will  be  useful  in 
predicting  performance  in  all  Navy  ratings,  but  at  present,  no  systematic  method  exists 
for  predicting  in  which  ratings  such  spatial  and  perceptual  tests  will  or  will  not  be  useful. 
There  is  a  need  for  a  broad-based  taxonomy  of  Navy  ratings  that  groups  ratings  according 
to  their  ability  and  skill  requirements,  and  that  considers  abilities  and  personal  character¬ 
istics  in  addition  to  those  covered  by  traditional  tests  of  cognitive  ability. 

Although  the  rating  serves  as  the  unit  of  analysis  for  this  research,  the  skills, 
abilities,  etc.  required  to  perform  successfully  in  that  rating  are  referred  to  as  the  job 
requirements.  Since  "rating"  and  "job"  are  not  synonymous,  this  usage  is  somewhat 
imprecise.  There  are,  however,  several  reasons  to  prefer  the  term  "job  requirements"  to 
terms  such  as  "requirements  of  the  rating."  First,  ratings  consist  of  groups  of  highly 
similar  jobs.  Thus,  one  can  argue  that  job  requirements  of  the  most  general  sort  are 
rating  requirements.  Second,  the  term  "job  requirements"  is  widely  used  and  understood 
whereas  the  term  "rating"  has  a  special  meaning  that  is  particular  for  the  Navy. 
Throughout  this  report,  the  term  "job  requirement"  is  used  to  represent  skills,  abilities, 
knowledge,  or  experiences  needed  to  perform  successfully  in  most  of  the  jobs  that 
comprise  a  particular  rating. 

Purpose 


The  purpose  of  the  research  described  in  this  report  is  to  develop  a  practical  method 
for  classifying  Navy  ratings  on  the  basis  of  common  job  requirements.  The  focus  of  the 
present  study  is  on  job  requirements  for  enlisted  personnel  below  the  grade  of  Chief  Pettv 
Officer.  6  y 

A  taxonomy  based  on  job  requirements  would  be  useful  in  several  ways.  First,  as 
noted  earlier,  such  a  taxonomy  would  provide  a  basis  for  predicting  the  validity  of  new 
tests  and  would  provide  guidance  in  designing  validity  studies.  For  example,  the  validity 
of  tests  of  specific  abilities  (e.g.,  spatial  ability)  may  vary  systematically  across  ratings. 
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A  taxonomy  based  on  job  requirements  would  help  indicate  in  which  ratings  high  or  low 
validity  coefficients  might  be  expected.  Second,  such  a  taxonomy  might  be  useful  in 
evaluating  requests  for  transfer  from  one  rating  to  another.  The  outcome  of  transfers 
between  ratings  that  share  common  job  requirements  should  be  more  predictable  than 
transfers  that  involve  fundamental  changes  in  job  requirements. 

The  method  described  in  this  report  involves  grouping  ratings  on  the  basis  of  the 
variables  most  likely  to  predict  successful  performance  in  the  jobs  that  comprise  a  rating. 
Thus,  the  domain  of  this  taxonomy  is  potentially  broader  than  the  domain  of  aptitude  and 
skill  requirements.  In  particular,  the  taxonomy  constructed  here  considered  broad 
dimensions  such  as  performance  in  training  and  previous  job  experience  as  well  as 
traditional  cognitive  and  psychomotor  ability  dimensions.  This  taxonomy  did  not  consider 
other  predictors,  such  as  personality,  interests,  motivation,  or  life  history  measures  for 
reasons  that  are  discussed  later.  The  inclusion  of  training  and  experience  dimensions  as 
well  as  ability  dimensions  allowed  us  to  discriminate  between  three  types  of  ratings:  (1) 
those  in  which  traditional  cognitive  abilities  are  the  most  important  determinant  of 
success,  (2)  those  in  which  selected  abilities  that  are  not  highly  related  to  general  mental 
ability  (e.g.,  perceptual  motor  abilities)  are  important  determinants  of  success,  and  (3) 
those  in  which  individual  differences  in  ability  are  not  seen  as  major  determinants  of 
success. 


METHOD 

Although  several  methods  exist  for  developing  job  taxonomies  (Fleishman  & 
Quaintance,  1984),  none  of  these  methods  is  entirely  adequate  for  the  task  at  hand.  A 
major  task  in  the  present  study  was  therefore  to  develop  a  practical  method  for  forming  a 
predictor-based  taxonomy  of  Navy  ratings.  To  accomplish  this  end,  several  different 
methods  of  data  collection  and  of  defining  and  establishing  job  requirements  were 
assessed  using  a  limited  sample  of  Navy  ratings.  Thus,  the  present  study  entailed  five 
steps:  (1)  determining  dimensions  for  describing  job  requirements,  (2)  selecting  a  sample 
of  ratings  for  study,  (3)  establishing  job  requirements,  using  multiple  methods,  in  each 
rating,  (4)  assessing  convergence  between  methods,  and  (5)  classifying  ratings  on  the  basis 
of  job  requirements. 

Determining  Job  Requirements  Dimensions 

Job  requirements  could  be  phrased  in  terms  ranging  from  the  most  general  (e.g.,  good 
attitude  required)  to  the  most  specific  (e.g.,  ability  to  operate  specific  piece  of  equipment 
required).  In  defining  job  requirement  dimensions,  four  factors  were  considered.  First, 
dimensions  should  be  sufficiently  broad  to  apply  to  several  different  ratings.  Thus,  ability 
to  perform  specific  tasks  or  to  operate  specific  equipment  were  not  considered  in  forming 
a  rating  taxonomy.  Second,  dimensions  should  be  sufficiently  broad  to  be  assessed  by 
several  different  methods.  Thus,  job  requirement  dimensions  that  were  unique  to  a  single 
method  of  measurement  (e.g.,  choice  reaction  time,  criterion-keyed  biodata  blanks)  were 
not  included  in  the  taxonomy.  Third,  we  considered  the  "track  record"  of  different  job 
requirement  dimensions.  Only  those  dimensions  that  had  been  clearly  shown  in  the 
research  literature  to  be  related  to  job  performance  were  included  in  the  development  of 
the  rating  taxonomy.  Finally,  the  application  potential  of  different  job  dimensions  was 
considered.  For  example,  it  is  conceivable  that  several  physiological  measures  are  related 
to  performance  in  some  jobs,  but  it  is  doubtful  that  the  technology  exists  to  take 
advantage,  on  any  large  scale,  of  this  relationship.  Similarly,  there  is  some  evidence  that 
personality  variables  and  interests  re  related  to  performance  in  some  jobs,  but  the 
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research  evidence  is  inconsistent  and  generally  unencouraging  (Schmitt,  Gooding,  Noe,  & 
Kirsch,  1984);  the  application  potential  of  these  tests  is  therefore  in  doubt.  Only  those 
dimensions  that  could  be  reliably  and  economically  measured  and  that  presented  clear 
application  potential  were  including  the  taxonomy. 

The  criteria  listed  above  led  us  to  eliminate  several  theoretically  interesting 
dimensions  such  as  personality,  interests,  or  social  skills.  Since  this  report  deals  with  a 
predictor-based  taxonomy;  these  dimensions  were  eliminated. 

Five  job  requirement  dimensions  were  selected  to  form  the  basis  of  the  rating 
taxonomy.  They  are: 

!•  Cognitive  ability;  General  intelligence  and  closely  related  verbal,  mathematical, 
and  reasoning  abilities  . 

2-  Psychomotor  ability:  Abilities  related  to  visualization,  spatial  orientation,  and 
precision  of  motor  responses. 

3.  Information-processing  ability:  Abilities  involving  basic  information  processing 
activities,  such  as  selective  attention  or  time  sharing 

4.  Training:  Performance  in  structured  job  training  activities. 

5.  Experience:  Previous  work  history,  as  well  as  life  history  data  that  are  related 
to  job  performance. 

Selecting  a  Sample  of  Ratings 

Rather  than  apply  several  experimental  methods  to  all  of  the  ratings  in  the  Navy,  we 
selected  a  sample  of  approximately  one-fifth  of  the  current  ratings  for  detailed  study. 
Several  factors  were  considered  in  selecting  ratings.  First,  for  practical  reasons,  we 
wished  to  study  ratings  in  which  there  were  large  numbers  of  incumbents  centrally 
located.  Second,  we  wished  to  cover  several  major  job  families  by  sampling  ratings  that 
involved  clerical  work,  bench  work,  structural  work,  information-processing  work,  and 
work  with  people.  In  addition,  we  wished  to  cover  jobs  that  were  widely  representative  of 
Navy  activities  at  ship-based,  land-based,  and  aviation-related  sites.  These  considerations 
led  to  the  selection  of  the  18  ratings  listed  in  Table  1  as  the  focus  of  our  study. 

Establishing  Job  Requirements 

Review  of  research  in  personnel  selection  and  classification  (c.f.  Fleishman,  1975; 
Fleishman  &  Quaintance,  1984)  suggests  that  several  methods  are  commonly  used  to 
establish  the  extent  to  which  each  of  the  job  requirement  dimensions  described  earlier  are 
related  to  successful  performance  in  different  ratings.  Four  of  these  methods  were  used 
in  this  study:  (1)  occupational  ability  pattern  analysis,  (2)  expert  judgment,  (3) 
validation  research,  and  (4)  task  analysis.1  Expert  judgment  required  field  interviews 


Additional  methods  for  determining  job  requirements  include  worker-oriented  job 
analysis  questionnaires  (Fine,  1955;  McCormick,  3eanneret,  <3c  Mecham,  1972)  and  trait- 
oriented  methods  (Lopez,  Kesselman,  &  Lopez,  1981;  Primoff,  1974).  Each  of  these 
methods  requires  supervisors  or  job  experts  to  respond  to  extensive  questionnaires  or 
inventories,  and  was  judged  impractical  for  the  current  project. 
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with  subject-matter  experts;  the  other  three  methods  were  archival,  in  the  sense  that 
they  involved  synthesis  of  data  that  had  already  been  collected  for  other  purposes. 


Table  1 

Eighteen  Ratings  Selected  for  Study 


AC  Air  Traffic  Controller 

AM  Aviation  Structural  Mechanic 

AQ  Aviation  Fire  Control  Technician 

AZ  Aviation  Maintenance  Administrationman 

BM  Boatswain's  Mate 

CT  Cryptologist  Technician 

DS  Data  Systems  Technician 

EM  Electrician's  Mate 

ET  Electronics  Technician 

HM  Hospital  Corpsman 

IS  Intelligence  Specialist 

MA  Master  at  Arms 

MM  Machinists  Mate 

PC  Postal  Clerk 

RM  Radioman 

SH  Ship's  Serviceman 

ST  Sonar  Technician 


Occupational  Ability  Pattern  Analysis 

This  first  strategy  for  establishing  job  requirements  takes  advantage  of  the  extensive 
research  on  the  ability  and  training  requirements  for  civilian  jobs  that  is  catalogued  in  the 
Dictionary  of  Occupational  Titles  (DOT).  In  addition  to  job  descriptions,  the  provides 
DOT  estimates  of  the  extent  to  which  each  of  the  11  abilities  measured  by  the  General 
Aptitude  Test  Battery  (GATB)  are  required  for  each  of  several  thousand  civilian  jobs 
(Field  &  Field,  1977).  These  GATB  profiles  form  what  will  be  referred  to  here  as 
occupational  ability  patterns. 

The  method  of  occupational  ability  pattern  analysis  involves  matching  Navy  ratings 
to  civilian  equivalents  that  are  described  in  the  DOT.  This  was  achieved  by  using  the 
Military  Occupational  an  Training  Data  (MOTD)  manual,  which  lists  civilian  equivalents 
(by  DOT  number)  for  most  of  the  occupations  in  the  services.  Using  the  MOTD  (1985),  we 
were  able  to  identify  civilian  equivalents  for  17  of  the  18  ratings  studied;  there  is  no 


4 


civilian  equivalent  for  Sonar  Technician.  For  several  ratings,  more  than  one  civilian 
equivalent  could  be  identified.  For  example,  five  different  civilian  jobs  were  identified  as 
equivalent  to  the  rating  Master  at  Arms;  these  are  listed  in  Table  2.  The  number  of 
civilian  equivalents  identified  for  the  17  ratings  ranged  from  one  to  eight;  an  average  of 
four  civilian  equivalents  was  identified  for  each  Navy  rating. 


Table  2 

Civilian  Equivalents  to  Master  at  Arms 


Civilian  Equivalents 

DOT  # 

Guard,  Security 

372667034 

Merchant  Patroller 

372667038 

Police  Officer  I 

375263014 

State  Highway  Police  Officer 

375263018 

Sheriff,  Deputy 

377263010 

In  addition  to  GATB  ability  requirements,  estimates  of  the  level  of  reasoning, 
mathematical,  language,  and  specific  vocational  training  required  in  each  civilian  job 
were  obtained  from  the  DOT.  For  the  15  ratings  in  which  more  than  one  civilian 
equivalent  was  identified,  GATB  scores  and  training  requirement  scores  were  pooled  over 
all  civilian  equivalents  by  taking  the  mode  of  each  of  the  ability  and  training  scores  to 
form  an  occupational  ability  profile  for  that  rating.  Thus,  the  ability  profile  for  the 
rating  Master  at  Arms  is  an  average  of  the  ability  profiles  for  the  five  civilian  jobs  listed 
in  Table  2.2 

Occupational  ability  patterns  for  the  17  ratings  are  listed  in  Table  3.  GATB  ability 
requirements  and  training  requirements  are  scored  on  a  scale  from  1  to  5,  where  a  high 
score  indicates  that  a  high  level  of  that  ability  is  required  on  the  job;  an  exception  is 
specific  vocational  training  requirements,  which  were  scored  on  a  nine-point  scale. 

Expert  Judgment 

The  second  method  used  to  determine  job  requirements  involved  the  use  of  subject- 
matter  experts.  Specifically,  several  chiefs  (E-7,  E-S,  or  E-9)  in  each  of  the  18  ratings 
were  asked  to  judge  the  importance  of  several  abilities,  as  well  as  experience  and 
training,  in  determining  successful  performance  on  the  job.  To  avoid  inflation  that  often 
accompanies  self-ratings,  chiefs  were  asked  to  rate  the  abilities  and  experiences 
necessary  for  successful  performance  at  the  E-4  and  E-5  level  (Petty  Officer,  3rd  and  2nd 
class)  in  their  ratings.  Chiefs  were  qualified  to  serve  as  experts  in  judging  the 
determinants  and  performance  at  this  level  for  two  reasons:  (1)  all  chiefs  have  served  at 
the  E-4  and  E-5  level  in  their  rating,  and  (2)  most  chiefs  supervise  the  work  of  E-4s  and 
E-5s. 


2 The  mode  was  used  rather  than  the  mean  or  median  because  of  the  highly  skewed 
distributions  typically  found  in  these  data. 
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Table  3 


Occupational  Ability  Patterns  for  17  Navy  Ratings 


Rating 

AC 

AM 

AQ 

AZ 

BM 

CT 

DS 

EM 

ET 

HM 

IS 

MA 

MM 

MT 

PC 

RM 

SH 

GATB  Abilities 

Intelligence 

4 

3 

3 

3 

3 

4 

4 

3 

3 

4 

4 

3 

3 

3 

3 

3 

3 

Verbal  ability 

3 

3 

3 

3 

2 

3 

3 

3 

3 

3 

4 

3 

3 

3 

3 

3 

2 

Numerical  ability 

3 

3 

3 

3 

2 

3 

3 

3 

3 

3 

4 

2 

3 

3 

3 

3 

2 

Spatial  ability 

2 

4 

3 

2 

3 

3 

4 

4 

3 

3 

3 

2 

4 

3 

2 

3 

2 

Form  perception 

3 

4 

2 

3 

3 

3 

4 

3 

4 

3 

3 

3 

4 

2 

3 

3 

2 

Clerical  perception 

3 

2 

2 

4 

2 

3 

2 

2 

2 

3 

3 

2 

2 

2 

4 

3 

3 

Motor  coordination 

2 

3 

3 

3 

3 

2 

3 

3 

3 

3 

2 

3 

3 

3 

3 

3 

3 

Finger  dexterity 

2 

3 

3 

3 

2 

2 

3 

3 

3 

3 

2 

2 

3 

3 

3 

4 

2 

Manual  dexterity 

2 

4 

3 

3 

3 

2 

4 

3 

4 

3 

2 

3 

3 

3 

3 

3 

2 

Eye-hand-foot  coordination 

2 

2 

2 

1 

3 

1 

1 

1 

1 

2 

1 

3 

1 

2 

1 

1 

1 

Color  discrimination 

3 

1 

2 

1 

2 

2 

2 

3 

3 

2 

2 

1 

1 

2 

1 

1 

2 

Training  Required 

Reasoning 

5 

4 

4 

3 

3 

4 

4 

4 

4 

4 

5 

3 

4 

4 

4 

4 

3 

Mathematical 

4 

3 

4 

2 

2 

4 

4 

3 

4 

4 

4 

2 

4 

4 

2 

3 

3 

Language 

4 

3 

4 

3 

2 

4 

4 

3 

3 

4 

4 

3 

3 

4 

3 

4 

2 

Specific  vocational  training 

8 

7 

6 

4 

5 

7 

6 

7 

7 

6 

7 

4 

7 

6 

4 

7 

4 

Expert  judgments  were  obtained  in  field  interviews  conducted  in  Norfolk,  VA  and  San 
Diego,  CA.  Sample  sizes  for  the  two  locations  are  listed  in  Table  4.  With  the  exception 
of  the  AQ  rating,  we  were  able  to  interview  a  minimum  of  four  chiefs  in  each  rating.  We 
interviewed  chiefs  in  both  Norfolk  and  San  Diego  in  two  ratings  (AC  and  MA)  to  assess  the 
similarity  in  responses  obtained  in  the  two  coasts.  In  general,  the  ability  profiles  obtained 
from  the  two  coasts  for  these  ratings  were  sufficiently  similar  to  be  pooled  into  a  single 
profile  for  each  rating.  That  is  within  ratings,  there  were  no  significant  differences 
between  the  Norfolk  and  San  Diego  samples  in  any  of  the  judgmental  indices  obtained 
from  subject-matter  experts. 


Table  4 

Expert  Judgment  Sample 


Rating 

Norfolk 

San  Diego 

Total 

AC 

5 

7 

12 

AM 

- 

6 

6 

AQ 

- 

2 

2 

AZ 

5 

- 

5 

BM 

- 

6 

6 

CT 

- 

6 

6 

DS 

4 

- 

4 

EM 

4 

- 

4 

ET 

4 

- 

4 

HM 

6 

- 

6 

IS 

- 

5 

5 

MA 

4 

3 

7 

MM 

5 

- 

5 

MT 

5 

- 

5 

PC 

5 

- 

5 

RM 

- 

8 

8 

SH 

- 

6 

6 

ST 

- 

5 

5 

In  addition  to  unstructured  interviews,  chiefs  responded  to  two  questionnaires 
designed  to  assess  the  extent  to  which  general  ability,  specific  cognitive  and  psychomotor 
abilities,  experience,  and  training  contribute  to  successful  performance.  The  first 
questionnaire  asked  them  to  indicate  the  importance  of  21  abilities  taken  from  the 
Fleishman  taxonomy  (Fleishman  &  Quaintance,  1984),  using  a  five-point  Likert-type  scale. 
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High  scores  indicate  that  the  ability  is  important.  The  second  questionnaire  presented  a 
policy  capturing  task,  in  which  subjects  judged  the  likelihood  of  success  of  hypothetical 
individuals  who  were  described  in  terms  of  their  intelligence,  performance  in  training, 
experience,  and  psychomotor  ability.  Cue  values  were  normally  distributed  and  mutually 
orthogonal,  allowing  us  to  determine  the  independent  impact  of  each  of  the  four  cues  on 
overall  judgment.  In  particular,  the  correlation  between  the  values  of  each  cue  and 
subjects'  judgment  of  the  50  profiles  served  as  an  index  of  the  relative  importance  of  that 
cue  in  judging  the  profiles.  The  larger  the  correlation  between  a  particular  cue  (e.g., 
intelligence)  and  judgments,  the  more  important  that  cue  was  in  subjects'  judgments 
regarding  probable  success  on  the  job.  The  two  questionnaires  used  to  obtain  expert 
judgments  are  presented  in  Appendices  A  and  B. 


In  summary,  each  expert  provided  estimates  of  the  importance  of  21  abilities;  policy 
capturing  indices  of  the  importance  of  intelligence,  training,  experience,  and  psychomotor 
ability  were  also  obtained  for  each  expert.  These  data  were  obtained  from  2  to  12  experts 
in  each  rating  within  each  rating;  these  scores  were  pooled  across  experts  to  form  an 
unweighted  average  expert  judgment  profile  for  each  of  the  18  ratings  included  in  this 
study.  Average  expert  judgments  profiles  for  each  rating  are  presented  in  Table  5. 


Validation  Research 


The  third  method  used  to  determine  job  requirements  involved  a  review  of  recent 
validation  research.  Although  research  in  validity  generalization  suggest  that  much  of  the 
variability  in  the  validity  of  cognitive  ability  tests  is  artifactual,  there  is  reason  to 
believe  that:  (1)  there  is  some  systematic  variability  in  the  validity  of  different  types  of 
predictors  within  job  families,  (2)  there  is  systematic  variability  in  the  validity  of 
different  predictors  across  job  families,  and  (3)  differences  in  the  validity  of  predictors 
reflect  in  part  the  relative  importance  of  different  abilities  in  determining  job  perfor¬ 
mance  (see  Fleishman,  1975;  Ghiselli,  1966;  Gutenberg,  Arvey,  Osburn,  &  (Jeanneret,  1983; 
Hunter  &  Hunter,  1984).  For  example,  if  general  intelligence  is  important  in  job  A  and 
ess  important  in  job  B,  we  should  expect  that  cognitive  ability  tests  will  show  higher 
levels  of  validity  in  job  A  than  in  job  B.  6 


Because  research  on  paper  and  pencil  cognitive  ability  tests  has  been  reviewed 
extensively  in  the  validity  generalization  literature,  our  review  concentrated  on  validity 
studies  that  examined  alternatives  to  paper-and-pencil  tests.  We  classified  the  results  of 
these  validity  studies  according  to  two  criteria,  predictor  type  and  job  family.  The 
predictors  reviewed  were:  (1)  cognitive  ability  tests,  (2)  physical  and  psychomotor  ability 
tests,  (3)  training  measures,  (4)  work  history  measures,  and  (5)  life  history  (biodata) 
measures.  Note  that  these  five  classes  of  predictors  were  similar  but  not  identical  to  the 
five  dimensions  chosen  to  form  a  basis  for  the  taxonomy.  These  five  dimensions 
correspond  to  those  most  often  used  in  classifying  validation  studies.  Validation  results 
were  also  classified  according  to  six  of  the  major  job  families  included  in  the  DOT:  (1) 
professional,  technical,  and  managerial,  (2)  clerical  and  sales,  (3)  service  occupations,  (4) 
machine  trade  occupations,  (5)  structural  work  occupations,  and  (6)  miscellaneous 
occupations.  Some  of  the  cells  of  the  5x6  predictor  type  x  job  family  table  contained 
more  than  one  validity  coefficient;  unweighted  mean  validity  coefficients  were  computed 
(using  Fisher's  transformation)  for  each  cell  with  multiple  entries.  The  results  of  our 
review,  classified  by  predictor  type  and  job  family,  are  presented  in  Table  6. 

We  were  able  to  sort  the  civilian  equivalent  of  the  17  Navy  ratings  into  one  or  more 
of  the  job  families  listed  in  Table  6,  and  thus  were  able  to  obtain  estimates  of  the  validity 
of  these  five  classes  of  tests  in  predicting  performance  in  each  rating.  Where  civilian 
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Table  5 


Mean  Expert  Judgments  by  Rating 


Rating 


AC 

AM 

AQ 

AZ 

BM 

CT 

DS 

EM 

ET 

HM 

IS 

MA 

MM 

MT 

PT 

RM 

5H 

ST 

Abilities  from  Fleishman 
Taxonomy 

Oral  comprehension 

5.0 

3.8 

3.5 

4.4 

4.5 

4.1 

3.7 

3.0 

3.5 

4.1 

4.4 

4.2 

3.6 

4.4 

4.4 

4.1 

3.5 

4.0 

Written  comprehension 

4.2 

3.6 

4.0 

4.6 

3.8 

4.0 

4.7 

3.7 

4.2 

4.5 

4.2 

4.4 

3.8 

4.4 

4.4 

4.4 

3.8 

4.2 

Oral  expression 

4.9 

3.6 

3.5 

3.6 

4.6 

4.1 

3.5 

3.7 

3.7 

4.0 

3.8 

4.4 

4.0 

3.8 

4.2 

3.5 

4.0 

4.2 

Written  expression 

3.0 

3.5 

3.0 

3.0 

3.1 

4.1 

3.5 

3.0 

3.2 

3.8 

3.6 

4.4 

3.8 

3.2 

3.8 

3.1 

3.8 

3.6 

Memorization 

4.S 

3.3 

4.0 

3.8 

3.8 

4.6 

4.0 

3.5 

4.5 

4.1 

4.0 

4.0 

4.8 

4.2 

4.2 

4.0 

3.6 

4.8 

Problem  sensitivity 

4.6 

4.1 

4.0 

2.8 

4.5 

4.0 

4.7 

4.2 

4.2 

3.8 

3.4 

3.7 

4.8 

4.6 

4.0 

4.0 

3.3 

4.8 

Mathematical  reasoning 

2.2 

2.5 

4.5 

2.0 

3.1 

2.1 

4.5 

2.2 

4.0 

2.6 

3.6 

1.7 

2.2 

2.2 

3.6 

1.2 

2.8 

3.8 

Number  facility 

2.3 

2.1 

4.5 

3.2 

2.6 

1.6 

3.7 

2.0 

3.7 

2.3 

3.6 

1.5 

2.2 

2.0 

4.6 

1.6 

4.6 

4.0 

Deductive  reasoning 

4.3 

3.1 

5.0 

2.8 

4.0 

3.8 

4.7 

4.0 

4.5 

2.8 

3.4 

3.8 

4.0 

4.4 

2.8 

3.3 

3.5 

4.4 

Inductive  reasoning 

3.7 

3.3 

5.0 

3.4 

4.1 

3.8 

4.5 

3.5 

4.2 

3.6 

3.6 

4.0 

4.0 

3.6 

3.2 

3.6 

3.5 

4.4 

Speed  of  closure 

4.6 

3.1 

3.0 

1.6 

4.0 

3.6 

3.7 

2.0 

4.2 

3.5 

3.4 

3.4 

2.8 

3.4 

3.2 

2.6 

2.1 

4.4 

Flexibility  of  closure 

3.4 

3.6 

3.5 

2.4 

3.0 

3.5 

3.2 

2.2 

3.5 

2.8 

3.6 

3.2 

2.0 

2.8 

2.6 

2.3 

2.8 

3.8 

Spatial  orientation 

4.2 

3.5 

2.5 

1.4 

4.5 

2.6 

2.2 

1.2 

3.0 

2.5 

3.6 

3.1 

3.2 

2.8 

2.2 

1.3 

2.5 

3.6 

Visualization 

3.7 

4.0 

3.5 

1.6 

3.1 

1.8 

3.0 

2.0 

2.7 

2.1 

3.4 

3.4 

3.8 

3.6 

2.2 

1.6 

3.5 

3.2 

Perceptual  speed 

3.1 

4.0 

3.5 

2.2 

3.5 

3.1 

3.0 

1.7 

3.7 

3.0 

4.2 

3.4 

3.6 

3.4 

2.6 

3.5 

3.5 

4.6 

Control  precision 

3.0 

3.6 

5.0 

1.6 

4.1 

3.0 

2.5 

2.5 

3.5 

2.6 

2.0 

3.1 

4.2 

4.6 

2.6 

3.2 

2.6 

4.0 

Selective  attention 

4.5 

4.1 

5.0 

4.0 

4.1 

4.6 

3.0 

3.5 

4.0 

3.6 

3.5 

3.2 

4.2 

4.8 

4.2 

4.3 

3.0 

4.0 

Time  sharing 

4.7 

4.0 

4.0 

3.8 

3.5 

4.6 

3.7 

3.0 

4.0 

4.0 

3.5 

3.7 

4.0 

4.2 

4.4 

4.2 

3.0 

4.0 

Auditory  attention 

4.8 

2.6 

3.0 

3.6 

4.0 

4.8 

2.7 

1.7 

3.5 

3.3 

2.2 

3.2 

3.8 

4.4 

2.6 

3.2 

2.8 

4.8 

Speech  hearing 

4.8 

3.6 

3.0 

3.0 

4.1 

3.6 

2.5 

2.5 

3.2 

4.0 

3.0 

3.0 

4.0 

2.6 

3.0 

3.2 

4.0 

3.6 

Speech  clarity 

4.9 

3.8 

3.0 

4.0 

4.3 

3.6 

3.5 

3.0 

3.7 

4.0 

4.2 

4.8 

4.6 

4.0 

4.4 

3.8 

3.6 

3.8 

Policy  Capturing  Indices 

Intelligence 

.17 

.22 

.27 

.49 

.15 

.48 

.45 

.30 

.35 

•33 

.47 

.27 

.17 

.18 

.38 

.34 

.31 

.24 

Training 

.31 

.35 

.37 

.32 

.37 

.39 

.41 

.50 

.47 

.4  7 

.40 

.29 

.40 

.38 

.42 

.40 

.45 

.44 

Experience 

.22 

.32 

.32 

.25 

.20 

.15 

.14 

.25 

.18 

.18 

.16 

.26 

.32 

.20 

.20 

.23 

.26 

.25 

Perceptual/motor  skills 

.50 

.38 

.38 

.09 

.51 

.19 

.20 

.22 

.28 

.28 

.09 

.28 

.38 

.47 

.16 

.24 

.15 

.43 

Table  6 


Estimated  Validity  of  Five  Classes  of  Predictors 
in  Six  Job  Families3 


Job 

Family 

Cognitive 

Ability 

Physical 

Ability 

Training 

Measures 

Work 

History 

Life 

History 

Professional 

Technical,  managerial 

.48 

.26 

.15 

.15 

.31 

Clerical  and  sales 

.43 

.25 

.13 

.14 

.51 

Service  occupation 

.33 

.26 

.13 

.14 

.32 

Machine  trade  occupation 

.45 

.34 

.11 

.11 

.13 

Structural  work  occupations 

.37 

.40 

.13 

.14 

.32 

Miscellaneous  occupations 

.30 

.44 

.13 

.14 

.32 

3 

Articles  included  in  review  were  Ash  and  Levine  (1985);  Gordon  and  Kleinman  (1976); 
Hough,  Keyes,  and  Dunnette  (1983);  Hunter  and  Hunter  (1984);  Latham  and  Saari  (1984); 
Love  (1981);  Mount,  Muchinsky,  and  Hanser  (1977);  Reilly  and  Chao  (1982>,  and  Schmidt, 
Hunter,  and  Pearlman  (1981).  Several  of  these  studies  were  in  turn  reviews  of  current 
research. 


equivalents  of  a  particular  rating  could  be  matched  to  multiple  job  families  (e.g.,  clerical 
and  service  occupations),  we  computed  the  unweighted  means  of  the  validity  coefficients 
from  those  families  to  represent  the  estimated  validity  of  each  test  type  for  that  rating. 

Task  Analysis 

Task  analysis  data  were  obtained  from  two  sources.  First,  detailed  task  analyses 
were  provided  for  each  rating  by  the  Navy  Occupational  Data  Analysis  (NODAC). 
NODAC  task  analyses  report  the  frequency  with  which  members  of  each  rating  do, 
supervise,  and  assist  in  performing  each  of  up  to  several  hundred  tasks,  and  the  frequency 
with  which  different  types  of  equipment  are  used;  the  exact  number  of  tasks  and  types  of 
equipment  varies  from  rating  to  rating.  Task  analysis  results  are  reported  separately  for 
each  enlisted  pay  grade,  allowing  analysts  to  examine  in  detail  changes  in  job  duties  as 
individuals  move  up  through  the  ranks.  Second,  higher-order  task  data  could  be  obtained 
from  the  Navy  Enlisted  Occupational  Standards.  These  standards  describe  specific 
responsibilities  assigned  to  each  rating;  each  of  these  standards  is  in  turn  linked  to  the 
performance  of  one  or  more  of  the  tasks  included  in  the  detailed  task  analyses. 

As  is  typical  of  task  analysis  data,  the  majority  of  the  tasks  and  the  occupational 
standards  associated  with  each  rating  were  unique  to  that  rating.  As  a  result  very  few 
tasks  or  standards  were  present  in  enough  ratings  to  yield  meaningful  comparisons  of  the 
different  ratings  in  terms  of  the  tasks  performed  or  the  responsibilities  assigned  to  each 
rating.  Rather  than  providing  a  well  defined  list  of  job  requirements  for  each  rating,  such 
as  those  presented  in  Tables  3  and  5,  task  analysis  data  were  used  to  obtain  a  better 
understanding  of  each  rating,  and  served  as  the  basis  for  our  initial  hypotheses  regarding 
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the  dimensions  that  might  be  used  to  classify  jobs.  On  the  basis  of  task  and  occupational 
standards  data  for  each  of  the  18  ratings,  three  general  classes  of  ratings  were 
hypothesized: 

1.  Ratings  in  which  cognitive  ability,  training,  and  experience  are  emphasized  as 
determinants  oi  job  performance. 

2.  Ratings  in  which  cognitive,  perceptual-motor,  and  information  processing 
abilities  are  emphasized  as  determinants  of  job  performance. 

3.  Ratings  in  which  training  and  experience  are  emphasized  as  determinants  of  job 
performance. 

Assessing  Convergence 

Convergence  could  be  assessed  in  two  ways.  First,  since  multiple  expert  judgments 
were  obtained  for  each  rating,  inter-rater  agreement  could  be  assessed.  Second,  the 
convergence  between  occupational  ability  patterns,  validity  profiles,  and  expert  judg¬ 
ments  profiles  was  assessed  by  correlating  the  results  of  these  methods  across  the  18 
ratings  studied.  For  example,  validity  studies,  expert  judgments  and  occupational  ability 
profiles  could  all  be  used  to  assess  the  importance  of  general  cognitive  ability  in  each 
rating.  By  correlating  estimates  from  each  method  across  the  18  ratings,  convergence  of 
three  methods  could  be  assessed. 


RESULTS 


Inter-rater  Agreement/Expert  3udgment 

Since  correlation  coefficients  are  strongly  affected  by  irrelevant  differences  in  the 
means  and  standard  deviations  of  each  rater's  judgment,  as  well  as  differences  in  the 
covariance  between  cues  and  judgments,  we  did  not  compute  inter-rater  agreement 
statistics  for  the  policy  capturing  outcomes.  Inter-rater  agreement  for  the  21  Fleishman 
ability  dimensions  was  indexed  in  two  ways.  First,  the  extent  to  which  the  18  ratings  can 
be  distinguished  from  one  another  on  each  of  the  21  ability  dimensions  provides  a  measure 
of  inter-rater  agreement  (Kavanaugh,  MacKinney,  <5c  Wolins,  1970).  By  measuring  the 
difference  between  ratings  on  each  dimensions  in  a  multi-rater  multi-dimension  design,  it 
is  possible  to  estimate  the  convergent  validity  of  expert  ratings.  Second,  a  concrete  index 
of  inter-rater  agreement  can  be  obtained  by  computing  the  standard  deviation,  within 
each  of  the  18  ratings,  for  each  of  the  21  ability  dimensions. 

The  data  showed  considerable  evidence  of  inter-rater  agreement.  First,  there  were 
significant  differences  between  the  mean  scores  of  the  18  ratings  on  17  of  the  21 
Fleishman  ability  dimensions;  F  statistics  for  these  17  dimensions  ranged  from  1.7  to  5.0, 
with  an  average  F  of  2.59. 3  Furthermore,  these  differences  between  ratings  were  large 
relative  to  the  variability  in  expert  judgment  within  each  rating.  On  the  average, 
differences  between  ratings  accounted  for  32  percent  of  the  variability  in  expert 
judgment;  the  average  between-rating  variance  was  2.5  times  as  large  as  the  average 
within-rating  variance.  Levels  of  convergent  validity  reported  here  are  comparable  with 
agreement  levels  reported  by  Borman  (1978)  for  laboratory  studies  of  rating  accuracy. 


3 

No  differences  were  found  for  the  importance  of  written  comprehension,  written 
expression,  memorization,  or  perceptual  speed. 
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The  average  within-rating  standard  deviation  was  .95.  This  indicates  that  on  average, 
experts  who  were  evaluating  the  importance  of  each  ability  dimensions  were  consistent,  in 
an  absolute  sense,  in  their  judgments.  In  concrete  terms,  this  figure  indicates  that  in  over 
85  percent  of  all  judgments,  experts  describing  the  same  rating  were  within  one  scale 
point  (a  five-point  scale  was  used)  of  the  mean  for  that  rating.  Taken  together,  the  inter¬ 
rater  agreement  data  cited  above  suggest  that  experts  were  able  to  agree  in  their 
judgments  regarding  the  ability  requirements  of  their  ratings. 

Convergence  Analysis 

In  all,  45  separate  measures  were  obtained  that  provided  information  regarding  job 
requirements.  Data  from  occupational  ability  analyses  included  for  training  requirement 
measures  and  11  GATB  ability  requirement  measures  were  obtained  from  the  DOT.  Our 
review  of  the  validation  research  yielded  five  validity  estimates  for  each  rating.  Expert’s 
judgment  data  yielded  21  ability  estimates  and  four  policy  capturing  measures.  To  reduce 
the  complexity  of  our  analysis,  composite  scores  were  formed  to  summarize  the  21  ability 
estimates  obtained  from  experts.  Separate  composites  were  formed  to  summarize  the  11 
GATB  requirement  scores  obtained  from  the  DOT. 

Composite  Scores 

Composite  scores  represented  unweighted  means  of  the  ability  estimates  obtained 
from  experts.  Three  composites  accounted  for  over  80  percent  of  the  variance  in  the  21 
ability  estimates.  They  were: 

1.  EXREAS:  A  reasoning  composite  made  up  of  problem  sensitivity,  deductive 
reasoning,  and  inductive  reasoning. 

2.  EXPERC:  A  perceptual  composite  made  up  of  flexibility  of  closure,  spatial 
orientation,  visualization,  conceptual  speed,  speech  hearing,  and  speech  clarity. 

3.  EXIP:  An  information  processing  composite  made  up  of  selective  attention,  time 
sharing,  auditory  attention,  and  speed  of  closure. 

Three  composites  accounted  for  over  85  percent  of  the  variance  in  the  GATB  requirement 
scores.  They  were: 

1.  GCOG:  A  cognitive  composite  consisting  of  intelligence,  verbal  ability,  and 
numerical  ability. 

2.  GPM:  A  perceptual-motor  composite  consisting  of  spatial  ability,  form 

perception,  motor  coordination,  finger  dexterity,  and  manual  dexterity. 

3.  GIP:  An  information  processing  composite,  which  included  only  the  clerical 
perception  test. 

Convergence  Between  Methods 


The  extent  to  which  methods  agreed  was  determined  by  correlating  the  data  obtained 
from  occupational  ability  patterns,  validation  research,  and  expert  judgment  over  the  18 
ratings.  Results  of  this  analysis  are  shown  in  Table  7;  only  those  correlations  that  are 
significant  at  the  .05  level  are  shown.  On  the  whole,  Table  7  does  not  indicate  extensive 
agreement  between  the  three  methods  analyzed.  However,  it  is  important  to  note  that 
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Table  7 


Convergence  Between  Methods 


a 


Occupational 
Ability  Pattern 

Validity 

Estimates 

Expert 
Judgment 
Policy  Capturing 

GCOG 

GPM 

GIP 

VCOGb 

VPHYS 

VTRAIN  V  WHIST 

VLHIST 

INTEL0 

TRAIN  EXPER 

PERMOT 

VCOG 

_ 

_ 

.55 

VPHYS 

-.76 

VWGUST 

- 

-.73 

.67 

VLHIST 

- 

-.73 

.90 

INTEL 

.48 

_ 

.54 

.45 

-.59 

—  - 

-.48 

TRAIN 

- 

- 

- 

- 

- 

- 

- 

EXPER 

- 

- 

- 

- 

- 

- 

PERMOT 

- 

-.52 

-.46 

.70 

- 

-.67 

EXREAS 

_ 

-.72 

.69 

_  __ 

-.68 

.45 

- 

.62 

EXPERC 

- 

_ 

- 

- 

- 

- 

-.55 

-.38 

.57 

EXIP 

- 

- 

- 

- 

- 

.68 

- 

- 

- 

.45 

Nonsignificant  correlations  are  not  shown.  Also,  correlations  between  measures  obtained  using  the  same  method  are 
not  shown. 

^Validity  estimates  for  cognitive  test,  physical  and  perceptual  motor  tests,  training  measures,  work  history,  and  life 
history  measures. 

CPolicy  capturing  indices  of  the  importance  of  intelligence,  training,  experience,  and  perceptual  motor  ability. 


because  of  the  small  sample  size  (N  =  18),  many  substantial  correlations  failed  to  reach 
conventional  levels  of  significance.  The  correlations  shown  in  Table  7  suggest  two 
conclusions:  (1)  results  of  the  expert  ability  ratings  and  policy  capturing  task  were  more 

(2)  data  from 
t  judgment  and 
.  First,  factor 

.. . .  ,  cvr>rTAC.  -  -  —  0-iieral  cognitive 

ability;  the  EXREA5  composite  was  correlated  with  the  policy  capturing  index  of  the 
importance  and  intelligence  for  performance,  and  both  of  these  were  correlated  with 
validation  research  results  for  cognitive  ability  tests.  Second,  directly  equivalent 
measures  were  obtained  from  the  three  methods  that  indicated  the  relevance  of  cognitive 
abilities  (GCOG,  VCOG,  INTEL)  and  perceptual-motor  abilities  (GPM,  VPHYS,  EXPER, 
PERMOT).  Expert  ratings  and  validity  studies  agreed  with  regard  to  the  importance  of 
cognitive  ability;  the  GATB  cognitive  ability  measure  was  correlated  with  the  policy 
capturing  index,  but  not  with  the  outcomes  of  validity  studies.  Expert  judgments  and 
validity  studies  also  agreed  with  regard  to  the  importance  of  perceptual  and  motor 
abilities;  the  GATB  perceptual-motor  composite  was  not  correlated  with  any  of  the  other 
perceptual-motor  measures. 

Taken  together,  the  results  shown  in  Table  1  suggest  that  data  obtained  from 
occupational  ability  patterns  did  not  converge  with  data  obtained  from  other  methods  of 
defining  job  requirements,  although  the  GATB  clerical  processing  test  (GIP)  showed 
convergence  with  several  other  measures.  Our  results  suggest  that  the  expert  judgment 
data  converge  adequately  with  data  obtained  from  validation  research.  We  therefore  used 
expert  judgment  data  as  the  primary  basis  for  grouping  ratings.  Once  groups  are  formed, 
it  is  possible  to  determine  whether  the  hypothesized  dimensions  that  were  identified  on 
the  basis  of  task  analysis  data  are  useful  in  grouping  ratings.  An  affirmative  answer  here 
would  indicate  some  level  of  convergence  between  task  analysis  and  expert  judgment 
data.  ° 

Grouping  Ratings 

Cluster  analysis  was  used  to  form  groups  of  ratings  that  were  homogenous  with 
regard  to  the  abilities  and  characteristics  rated  as  most  critical  for  successful  perfor¬ 
mance  on  the  job.  For  each  rating,  the  seven  measures  obtained  from  expert  judgments 
(three  composite  scores  and  four  policy  capturing  measures)  formed  an  ability  require¬ 
ment  profile.  Ratings  were  grouped  on  the  basis  of  profile  similarity,  using  the  minimum 
variance  method,  with  an  euclidean  distance  metric.  The  steps  in  the  iterative  cluster 
solution  are  shown  in  Figure  1. 

Several  criteria  were  used  in  evaluating  cluster  solutions.  First,  within-cluster 
variance  should  be  minimized.  Second,  the  number  of  clusters  should  be  minimized. 
Third,  there  should  be  no  more  than  one  outlier  (cluster  consisting  of  a  single  rating). 
Fourth,  the  between-cluster  variance  should  be  maximized.  These  criteria  were  met  best 
by  the  three  cluster  solution.  In  particular,  the  three  cluster  solution  showed  the  largest 
between-cluster  variance,  in  relation  to  the  within-cluster  variance,  of  any  of  the 
solutions  with  seven  or  fewer  clusters.4 


were  u  a  i  d  scores  ana 


compatiDie  wnn  validity  study  outcomes  than  _  _ 

occupational  ability  patterns  was  not  as  consistent  with  data  from  exper 
validation  research.  These  conclusions  are  based  on  several  considerations 
analytic  research  has  suggested  that  reasoning  is  a  central  facet  of  ppi 


4 Extracting  more  than  seven  clusters  results  in  multiple  outliers. 
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Number  of  Clusters 
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Figure  1.  Iterative  clustering  of  ratings. 
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The  nature  of  these  three  clusters  can  best  be  illustrated  by  plotting  the  profile  of 
scores  that  represent  the  cluster  centroids  (mean  score  on  each  dimension,  summing  over 
cluster  members).  For  the  purpose  of  comparison,  scores  on  the  seven  measures  used  to 
define  clusters  are  expressed  in  Z-score  terms.  Plots  of  the  three  cluster  centroids  are 
shown  in  Figure  2.  The  three  clusters  are  described  below: 

Cognitive  Cluster:  In  this  group  of  ratings,  general  intelligence  is  seen  as  the 
most  important  and  experience  and  perceptual  motor  skills  are  seen  as  the  least  important 
determinants  of  performance. 

2.  Practical  Ability  Cluster:  In  this  group  of  ratings,  general  intelligence  and 
training  are  seen  as  relatively  unimportant;  experience,  perceptual  motor  skills,  reason¬ 
ing,  and  information  processing  abilities  were  rated  as  more  important. 

3.  Training  Cluster:  In  this  group  of  ratings,  training  is  seen  as  most  important  and 
perceptual  motor  and  reasoning  abilities  are  seen  as  the  least  important  determinants  of 
performance. 

Cluster  membership  is  displayed  in  Table  8.  Cluster  members  are  listed  in  order  of  their 
similarity  (squared  euclidean  metric)  to  the  cluster  mean.  Thus,  Intelligence  Specialist  is 
most  representative  of  the  cognitive  cluster,  Machinist's  Mate  is  the  most  representative 
of  the  practical  ability  cluster,  and  Hospital  Corpsman  is  most  representative  of  training 
cluster. 

Convergence  Between  Expert  and  Task  Data 

As  noted  earlier,  although  task  analysis  data  did  not  provide  a  basis  for  analytic 
comparisons  of  the  ability  requirements  of  the  18  ratings  studied,  these  data  did  lead  to 
hypotheses  regarding  the  dimensions  along  which  ratings  might  be  grouped.  Close 
correspondence  was  found  between  the  hypothesized  dimensions  for  classifying  ratings  and 
the  actual  clusters  illustrated  in  Figure  2.  First,  we  hypothesized  that  one  group  of 
ratings  would  emphasize  cognitive  ability,  training,  and  experience.  The  cognitive  cluster 
emphasized  general  intelligence,  but  placed  less  emphasis  on  training  and  experience  than 
hypothesized.  Second,  we  hypothesized  that  one  group  of  ratings  would  emphasize 
cognitive,  perceptual  motor,  and  information  processing  abilities.  The  practical  ability 
cluster  emphasized  perceptual  motor  and  information  processing  abilities,  but  placed  less 
emphasis  on  general  intelligence  than  expected.  Finally,  we  hypothesized  that  one  cluster 
would  emphasize  training  and  experience.  The  training  cluster  emphasized  success  in 
training  but  placed  less  emphasis  on  experience  than  expected. 

Although  our  hypotheses  were  not  confirmed  in  all  details,  the  general  characteristics 
of  the  clusters  defined  on  the  basis  of  expert  judgment  data  were  highly  similar  to  those 
expected  on  the  basis  of  task  and  occupational  standards  data.  The  similarities  reported 
here  suggest  that  expert  judgment  data  and  task  analysis  data  lead  to  consistent 
conclusions  regarding  the  major  dimensions  for  classifying  Navy  ratings. 
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INTEL  TRAIN  EXPER  PERMOT 


EXPERC 


EXREAS  EXIPa 


Z  score 


■ - ■  Cognitive  Cluster 

V - $7  Practical  Ability  Cluster 

± ±  Training  Cluster 


Figure  2.  Three-cluster  solution 

aDimensions  are  policy  capturing  indices  of  intelligence,  training,  experience,  and 
perceptual  motor  ability  requirements  and  composite  abiity  ratings  for  the  perceptual, 
reasoning,  and  information  processing  composites. 
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Table  8 


Cluster  Membership  as  Determined  by 
Expert  Judgment  Data 


Cognitive  Cluster 

IS 

Intelligence  Specialist 

CT 

Cryptologist  Technician 

DS 

Data  Systems  Technician 

AZ 

Aviation  Maintenance  Administration 

Practical  Ability  Cluster 

MM 

Machinist's  Mate 

ST 

Sonar  Technician 

BM 

Boatswain's  Mate 

MT 

Missile  Technician 

AM 

Aviation  Structural  Mechanic 

AC 

Air  Traffic  Controller 

AQ 

Aviation  Fire  Control  Technician 

MA 

Master  at  Arms 

Training  Cluster 

HM 

Hospital  Corpsman 

PC 

Postal  Clerk 

RM 

Radioman 

ET 

Electronics  Technician 

SH 

Ship's  Serviceman 

EM 

Electrician's  Mate 

CONCLUSIONS 

1.  Navy  ratings  can  be  reliably  classified  in  terms  of  ability,  training,  and 
experience  requirements.  The  abilities  and  skills  necessary  for  successful  performance 
vary  from  job  to  job,  but  there  are  groups  of  ratings  that  exhibit  similar  patterns  of  job 
requirements.  The  ratings  sampled  in  this  study  were  chosen  to  represent  several  major 
facets  of  the  domain  of  all  Navy  ratings.  It  is  likely,  therefore,  that  the  classification 
system  developed  here  will  be  useful  when  applied  to  the  remaining  Navy  ratings. 
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2.  A  predictor-based  taxonomy  of  Navy  ratings  must  consider  both  traditional  and 
nontraditional  predictors  of  job  performance.  Traditional  predictors  include  cognitive 
ability  tests  (e.g.,  ASVAB)  and  training  measures.  To  adequately  classify  ratings, 
nontraditional  predictors  such  as  perceptual  motor  and  information  processing  tests  must 
also  be  considered.  Advances  in  computerized  testing  have  significantly  enhanced 
researchers'  capacity  to  measure  perceptual  and  information  processing  abilities.  The 
taxonomy  developed  here  suggests  that  tests  of  this  sort  will  be  useful  in  predicting 
performance  in  many,  but  not  all,  Navy  ratings. 

3.  The  rating  taxonomy  developed  here  leads  to  several  testable  hypotheses 
regarding  the  relative  usefulness  of  several  types  of  predictors  in  different  ratings.  An 
example  of  the  type  of  hypothesis  generated  by  this  taxonomy  is  presented  in  Table  9.  In 
general,  the  taxonomy  developed  here  suggests  that  the  predictive  validity  and  utility  of 
different  types  of  predictors  will  vary  across  groups  of  ratings.  More  important,  the 
optimum  battery  of  predictors  may  vary  systematically  across  ratings.  For  example, 
validity  generalization  research  suggest  that  cognitive  ability  tests  will  show  some  level 
of  validity  in  practically  all  ratings,  but  in  many  ratings  (i.e.,  those  in  the  practical  ability 
and  training  clusters),  cognitive  ability  tests  would  not  be  the  best  choice  for  predicting 
job  performance. 


Table  9 

Predicted  Usefulness  of  Several  Types  of  Predictors 


Representative 

Rating 

Cognitive 

Ability 

Tests 

Psychomotor  and 
Information  Processing 

Tests 

Training 

Measures 

MM 

Low 

High 

Moderate 

HM 

Moderate 

Low 

High 

IS 

High 

Low 

Moderate 

4.  A  predictor-based  taxonomy  has  practical  implications  for  the  design  and 
validation  of  selection  and  placement  tests.  For  example,  a  validity  study  that  assesses 
the  usefulness  of  a  new  computer-administered  information  processing  test  may  reach 
different  conclusions  if  conducted  in  a  sample  of  Machinist's  Mates  than  if  conducted  in  a 
sample  of  Hospital  Corpsman.  Studies  that  assess  the  comparative  validity  and  utility  of 
two  different  types  of  predictors  might  be  especially  sensitive  to  the  composition  of  the 
sample  employed.  In  general,  studies  that  are  designed  to  generalize  to  the  entire 
enlisted  population  should  include  representative  of  all  three  of  the  clusters  identified 
here. 

5.  Expert  judgment  provides  a  reliable  method  for  determining  ability,  training,  and 
experience  requirements  for  Navy  ratings.  Expert  judges  had  little  difficulty  compre¬ 
hending  the  tasks  presented  to  them,  and  were  able  to  discriminate  between  ratings  in 
terms  of  their  ability  requirements.  Expert  judges  had  little  difficulty  comprehending  the 
tasks  presented  to  them,  and  were  able  to  discriminate  between  ratings  in  terms  of  their 
ability  requirements.  Expert  judges  understood  the  Fleishman  ability  taxonomy  and  were 
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able  to  make  consistent  judgments  in  the  policy  capturing  task  (average  multiple  R, 
summing  across  judges,  was  in  excess  of  ,90).  Experts  showed  acceptable  levels  of  inteF- 
rater  agreement,  and  the  outcomes  of  the  two  expert  judgment  tasks  were  compatible 
with  data  from  validation  research  and  task  analyses. 

6.  Task  analysis  and  occupational  ability  pattern  analysis  are  not  useful  for 
determining  job  requirements  that  are  common  to  many  Navy  ratings.  The  majority  of 
the  task  data  supplied  for  each  rating  was  unique  to  that  rating.  Occupational  ability 
patterns  provided  a  potential  basis  for  grouping  jobs,  but  on  close  analysis  patterns  do  not 
provide  sufficient  discrimination  to  allow  the  grouping  of  ratings.  Occupational  ability 
P®.**®rns  tended  to  be  quite  flat,  and  were  highly  similar  across  ratings.  One  reason  for 
this  is  that  most  occupational  ability  patterns  represented  the  mean  of  several  civilian  job 
requirement  patterns.  It  is  generally  the  case  that  averaging  over  several  profile  tends  to 
produce  a  single  flat  profile  for  each  job;  this  same  phenomenon  was  observed  in  the 
present  study. 

Future  classification  of  Navy  ratings  should  take  into  account  personality  and 
interpersonal  relations  dimensions  that  are  related  to  job  success  as  well  as  ability, 
training,  and  experience  variables.  This  conclusion  is  based  on  our  field  interview  with 
subject-matter  experts.  Personality  dimensions  such  as  flexibility,  stress  tolerance,  and 
initiative  were  cited  by  many  experts  as  key  determinants  of  performance.  Although  data 
from  the  field  interviews  do  not  allow  detailed  analyses,  differences  between  ratings  were 
apparent  in  the  types  of  personality  dimensions  most  frequently  cited.  For  example,  in 
ratings  that  involved  customer  service  (e.g.,  SH,  PC),  ability  to  deal  effectively  with 
interpersonal  stress  was  cited  as  important,  whereas,  in  ratings  that  involved  independent 
decision  making  (e.g.,  AC),  mental  flexibility  was  cited  as  most  important.  Subject- 
matter  experts  agreed  that  personality  and  attitude  were  often  more  important  than 
ability  in  determining  success  or  failure  on  the  job. 

8.  The  rating  is  an  appropriate  unit  of  analysis  for  this  type  of  taxonomy.  Within 
ratings,  individual  jobs  can  vary  considerably.  For  example,  the  rating  IS  may  contain 
many  different  types  of  specialists.  Nevertheless,  there  is  enough  broad  similarity  in 
function  within  each  rating,  and  enough  difference  between  ratings  to  make  a  taxonomy 
of  ratings  both  plausible  and  useful. 


RECOMMENDATIONS 

1.  The  methods  developed  here  should  be  applied  to  the  remaining  Navy  ratings. 
Specifically,  expert  judgments  regarding  the  ability,  training,  and  experience  require¬ 
ments  of  each  rating  should  be  solicited.  Our  experience  in  collecting  data  for  this 
project  suggests  that  3  to  5  experts  should  be  sampled  in  each  rating,  and  that  it  may  be 
possible  to  conduct  at  least  some  of  this  data  collection  by  mail  rather  than  conducting 
field  interviews  for  every  rating.  Where  field  interviews  are  done,  it  is  possible  to  deal 
with  groups  of  practically  any  size  and  composition,  since  most  of  the  data  are  collected 
using  relatively  simple  questionnaires. 

2.  Predictions  that  are  generated  on  the  basis  of  the  taxonomy  presented  here 
should  be  tested.  As  adequate  operational  job  performance  measures  become  available  in 
different  ratings,  it  will  be  possible  to  assess  the  predictive  power  of  the  taxonomy 
developed  here.  In  particular,  research  on  the  comparative  validity  of  different 
predictors  in  different  ratings  is  recommended.  Although  the  taxonomy  developed  here 
does  allow  us  to  predict  the  rank-order  of  the  usefulness  of  predictors  in  different  ratings, 
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the  data  do  not  allow  us  to  predict  the  size  of  the  effect.  Thus,  it  is  not  currently 
possible  to  provide  explicit  estimates  of  the  actual  level  of  different  predictors  in 
different  ratings. 

3.  Alternatives  to  the  policy  capturing  task  used  in  the  present  study  should  be 
explored.  In  particular,  the  feasibility  of  obtaining  direct  estimates  of  the  importance  of 

intelligence,  training,  experiences,  and  perceptual  motor  ability  should  be  determined; 
such  a  task  would  be  considerably  simpler  than  the  current  policy  capturing  task,  and 
would  greatly  facilitate  the  use  of  surveys  rather  than  interviews  in  collecting  data  on  the 
remaining  ratings. 

4.  Personality  variables  and  interpersonal  skills  associated  with  successful  job 
performance  in  different  Navy  ratings  should  be  studied.  At  present,  little  is  known  about 
the  extent  to  which  personality  variables  affect  job  performance  in  the  Navy,  or  about  the 
particular  variables  that  are  most  strongly  related  to  job  performance.  The  addition  of 
personality  and  interpersonal  skill  information  may  allow  the  development  of  hierarchical 
taxonomies,  in  which  jobs  with  similar  ability  requirements  can  be  differentiated  in  terms 
of  their  personality  and  interpersonal  requirements. 
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APPENDIX  A 

ABILITY  RATING  QUESTIONNAIRE 


A-0 


QUESTIONNAIRE  1 


Instructions. 

A  set  of  abilities  have  been  listed  below.  We  are  interested 
in  your  judgments  about  the  importance  of  these  abilities  for 
successful  performance  in  jobs  similar  to  yours.  In  making  these 
judgments  think  about  what  E  4's  and  E  5's  in  your  rating  do. 

Think  about  an  individual  very  low  on  that  ability.  What  would  be 
the  consequences  for  successful  job  performance?  Use  the 
following  scale: 


1 

2 

3 

4 

5 

very  minimal 

small 

noticeable 

large 

very 

effect  on 

effect 

effect  on 

effect  on 

substantial 

performance 

on 

performance 

performance 

effect  on 

performance  performance 


Use  the  blank  provided  to  the  right  of  each  ability 
description  for  your  response. 

For  example,  if  a  radio  announcer  is  low  on  the  ability  of 
oral  expression,  the  consequences  for  successful  performance  on 
the  job  might  be  very  substantial.  Thus,  a  5  would  be  put  in  the 
blank  next  to  oral  expression.  Conversely,  if  a  radio  announcer 
is  low  on  the  ability  of  mathematical  reasoning,  the  consequences 
for  successful  job  performance  might  be  very  minimal,  and  a  1 
would  be  put  in  the  blank  for  mathematical  reasoning.  When 
answering,  keep  in  mind  that  we  are  most  interested  in  performance 
at  the  E4  and  E5  level. 
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1.  Oral  Caiprehension.  The  ability  to  understand  spoken  English 

words  and  sentences.  _ 

2.  Written  Ccnprehension.  The  ability  to  understand  written 

English  words  and  sentences.  _ 

3.  Oral  Expression.  The  ability  to  use  English  words  or  sentences 

in  speaking  so  others  will  understand.  _ 

4.  Written  Expression.  The  ability  to  use  English  words  or 

sentences  in  writing  so  others  will  understand.  _ 

5.  Memorization.  The  ability  to  remember  information,  such  as 

words,  nvnibers,  pictures,  and  procedures.  _ 

6.  Problem  Sensitivity.  The  ability  to  tell  when  something  is 

wrong  or  is  likely  to  go  wrong.  It  includes  being  able  to 
identify  the  whole  problem  as  well  as  the  elements  of  the 
problem.  _ 

7.  Mathematical  Reasoning.  The  ability  to  understand  and  organize 

a  problem  and  then  to  select  a  mathematical  method  of  formula 
to  solve  the  problem.  It  encompasses  reasoning  through 
mathematical  problems  to  determine  appropriate  operation  that 
can  be  performed  to  solve  problems.  The  actual  manipulation 
of  numbers  is  not  included  in  this  ability.  _ 

8.  Number  Facility.  The  degree  to  which  adding,  subtracting, 

multiplying,  and  dividing  dan  be  done  quickly  and  correctly. 
These  can  be  steps  in  other  operations  like  finding 
percentages  and  taking  square  roots.  _ 

9.  Deductive  Reasoning.  The  ability  to  apply  general  rules  to 

specific  problems  to  come  up  with  logical  answers.  It 
involves  deciding  if  an  answer  makes  sense.  _ 


10.  Inductive  Reasoning.  The  ability  to  combine  separate  pieces 

of  information,  or  specific  answers  to  problems,  to  form 
general  rules  or  conclusions.  It  involves  the  ability  to 
think  of  possible  reasons  for  why  things  go  together.  _ 

11.  Speed  of  Closure.  The  degree  to  which  different  pieces  of 

information  can  be  combined  and  organized  into  one  meaningful 
pattern  quickly.  It  is  not  known  beforehand  what  the  pattern 
will  be.  The  material  may  be  visual  or  auditory.  _ 

12.  Flexibility  of  Closure.  The  ability  to  identify  of  detect  a 

known  pattern  (like  a  figure,  word,  or  object)  that  is  hidden 
in  other  material.  _ 

13.  Spatial  Orientation.  The  ability  to  tell  where  you  are  in 

relation  to  the  location  of  some  object  or  to  tell  where  the 
object  is  in  relation  to  you.  _ 

14.  Visualization.  The  ability  to  imagine  how  something  will  look 

when  it  is  moved  around  or  when  its  parts  are  moved  or 
rearranged.  It  requires  the  forming  of  mental  images  of  how 
patterns  or  objects  would  look  after  certain  changes,  such  as 
unfolding  or  rotation.  _ 

15.  Perceptual  Speed.  The  degree  to  which  one  can  compare 

letters,  numbers,  objects,  pictures,  or  patterns,  quickly  and 
accurately.  This  ability  also  includes  comparing  a  presented 
object  with  a  remembered  object.  _ 


16.  Control  Precision.  The  ability  to  move  controls  of  a  machine 
or  vehicle.  This  involves  the  degree  to  which  these  controls 
can  be  moved  quickly  and  repeatedly  to  exact  positions.  _ 
Selective  Attention.  The  ability  to  concentrate  on  a  task  one 
is  doing.  This  ability  involves  concentrating  while 
performing  a  boring  task  and  not  being  distracted.  _ 

18.  Time  Sharing.  The  ability  to  shift  back  and  forth  between  two 

or  more  sources  of  information.  _ 

19.  Auditory  Attention.  The  ability  to  focus  on  a  single  source 

of  auditory  information  in  the  presence  of  other  distraction 
and  irrelevant  auditory  stimuli.  _ 

20.  Speech  Hearing.  The  ability  to  learn  and  understand  the 

speech  of  another  person.  _ 

21.  Speech  Clarity.  The  ability  to  communicate  orally  in  a  clear 

fashion  understandable  to  a  listener. 
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APPENDIX  B 

POLICY  CAPTURING  QUESTIONNAIRE 
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QUESTIONNAIRE  2 


Instructions. 

Different  abilities  and  characteristics  are  important  for 
succeeding  in  different  ratings  in  the  Navy.  We  will  describe  50 
individuals  in  terms  of  four  of  their  characteristics; 

General  Intelligence.  The  ability  to  reason,  to  learn,  and  to 
choose  appropriate  courses  of  action. 

Performance  in  Training.  The  performance  in  Navy  training 

courses  that  are  specifically  related  to  an  individual's 
current  job. 

Relevant  Work  Experience.  The  extent  to  which  prior  civilian 
and  Navy  jobs  involved  skills  or  knowledge  relevant  to 
the  individual's  current  job. 

Perceptual  and  Motor  Skills.  The  ability  to  make  quick, 

accurate  responses  to  small  events  in  the  environment, 
such  as  an  indicator  light  or  a  blip  on  a  radar  screen. 

Your  task  is  to  predict  how  successful  each  individual  might 
be  in  your  rating,  given  what  you  know  about  these  four 
characteristics.  For  example,  how  successful  do  you  think  a 
person  would  be  in  your  rating  if  that  person  had  done  very  well 
in  training,  but  had  very  poor  perceptual  and  motor  skills? 
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We  will  describe  each  one  of  these  four  characteristics  in 
terms  ranging  from  very  low  to  very  high.  Definitions  of  these 
terms  are  given  below: 

Very  Low  -  bottom  10%  in  the  Navy 
low  -  10^  -  30^percentile 

Average  -  middle  40%  of  the  Navy  (30^  -  70^  percentile) 
High  -  70^  -  90tllpercentile 
Very  High  -  top  10%  of  the  Navy 


We  will  ask  you  to  read  each  description,  and  then  rate  that 
individual's  chance  of  performing  successfully  on  the  job  using 
the  scale  below. 


1 _ 2 _ 3 

almost  likely  average 

certain  to  fail  chance 
to  fail  of  success 

on  the  job 


4 


likely 

to 

succeed 


_ 5 

almost 
certain  to 
succeed 


Please  remember  that  we  are  most  interested  in  performance  at 
the  E4  and  E5  level . 
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Individual  #  1 

General  Intelligence  -Very  Low 

Performance  in  Training  -High 

Relevant  Work  Experience  -  Low 

Perceptual  and  Motor  Skills  -High 

How  likely  Is  It  that  this  person  will  succeed? 
Individual  I  2 


General  Intelligence 

-  Average 

Performance  In  Training 

-High 

Relevant  Work  Experience 

-  High 

Perceptual  and  Motor  Skills 

-  Average 

How  likely  Is  It  that  this  person  will  succeed?_ 
Individual  #3 

General  Intelligence  -High 

Performance  In  Training  -Low 
Relevant  Work  Experience  -Very  High 
Perceptual  and  Motor  Skills  -Very  High 

How  likely  Is  It  that  this  person  will  succeed?_ 
Individual  14 

General  Intelligence  -Very  High 

Performance  in  Training  -Average 
Relevant  Work  Experience  -Average 
Perceptual  and  Motor  Skills  -Low 

How  likely  Is  it  that  this  person  will  succeed?^ 
Individual  #5 

General  Intelligence  -Average 

Performance  in  Training  -High 
Relevant  Work  Experience  -High 
Perceptual  and  Motor  Skills  -Average 

How  likely  is  it  that  this  person  will  succeed? 
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Individual  #  6 

General  Intelligence  -High 

Performance  In  Training  -  Average 
Relevant  Work  Experience  -  Very  Low 
Perceptual  and  Motor  Skills  -Average 

How  likely  is  It  that  this  person  will  succeed? 
Individual  #  7 

General  Intelligence  -  Very  Low 

Performance  in  Training  -Very  High 
Relevant  Work  Experience  -Average 
Perceptual  and  Motor  Skills  -Average 

How  likely  is  it  that  this  person  will  succeed?^ 
Individual  #  8 

General  Intelligence  -High 

Performance  in  Training  -Very  High 
Relevant  Work  Experience  -Low 
Perceptual  and  Motor  Skills  -Average 

How  likely  . is  It  that  this  person  will  succeed?^ 
Individual  #  9 


General  Intelligence 

-  Average 

Performance  in  Training 

-High 

Relevant  Work  Experience 

-Average 

Perceptual  and  Motor  Skills 

-High 

How  likely  is  it  that  this  person  will  succeed?^ 
Individual  #  10 

General  Intelligence  -Average 

Performance  in  Training  -Very  High 

Relevant  Work  Experience  -Very  High 

Perceptual  and  Motor  Skills  -Very  High 

How  likely  is  it  that  this  person  will  succeed? 
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Individual  #  11 

General  Intelligence  .  -  Low 

Performance  In  Training  -  Average 

Relevant  Work  Experience  -  Low 

Perceptual  and  Motor  Skills  -Low 

How  likely  is  It  that  this  person  will  succeed? 
Individual  #  12 


General  Intelligence 

-  Average 

Performance  in  Training 

-  High 

Relevant  Work  Experience 

-  High 

Perceptual  and  Motor  Skills 

-  High 

How  likely  is  it  that  this  person  will  succeed?^ 
Individual  #  13 

General  Intelligence  -Very  High 

Performance  in  Training  -Average 
Relevant  Work  Experience  -Low 

Perceptual  and  Motor  Skills  -Average 

How  likely  Is  It  that  this  person  will  succeed?^ 
Individual  # 14 

General  Intelligence  -High 

Performance  In  Training  -Very  Low 

Relevant  Work  Experience  -Very  Low 

Perceptual  and  Motor  Skills  -Very  Low 

How  likely  is  it  that  this  person  will  succeed?^ 
Individual  # 15 

General  Intelligence  -High 

Performance  in  Training  -Low 

Relevant  Work  Experience  -Very  High 

Perceptual  and  Motor  Skills  -tow 

How  likely  is  it  that  this  person  will  succeed? 
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Individual  I  16 


General  Intelligence  -Low 

Performance  in  Training  -Low 
Relevant  Work  Experience  -Average 
Perceptual  and  Motor  Skills  -Average 

How  likely  Is  It  that  this  person  will 
Individual  #  17 

General  Intelligence  -Average 

Performance  In  Training  -High 

Relevant  Work  Experience  -Average 

Perceptual  and  Motor  Skills  -High 

How  likely  is  it  that  this  person  will 
Individual  #18 

General  Intelligence  -Low 

Performance  in  Training  -Low 

Relevant  Work  Experience  -High 

Perceptual  and  Motor  Skills  -High 

How  likely  is  it  that  this  person  will 
Individual  #19 

General  Intelligence  -High 

Performance  in  Training  -Low 

Relevant  Work  Experience  -High 

Perceptual  and  Motor  Skills  -Average 

How  likely  is  it  that  this  person  will 
Individual  #20 


General  Intelligence 

-Average 

Performance  in  Training 

-Average 

Relevant  Work  Experience 

-Average 

Perceptual  and  Motor  Skills 

.Average 

How  likely  is  it  that 

this  person 
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succeed? 


succeed? 


succeed? 


succeed? 


succeed? 


Individual  #21 

General  Intelligence  -Low 

Performance  In  Training  -Average 

Relevant  Work  Experience  -High 

Perceptual  and  Motor  Skills  -Very  High 

How  likely  is  it  that  this  person  will  succeed? 
Individual  #22 

6eneral  Intelligence  -Average 

Performance  in  Training  -Average 

Relevant  Work  Experience  -Low 

Perceptual  and  Motor  Skills  -Average 

How  likely  is  it  that  this  person  will  succeed?^ 
Individual  #23 


General  Intelligence 

-Low 

Performance  in  Training 

-High 

Relevant  Work  Experience 

-Low 

Perceptual  and  Motor  Skills 

-Very  Low 

How  likely  is  it  that  this  person  will  succeed?^ 
Individual  #24 


General  Intelligence 

-High 

Performance  in  Training 

-Very  Low 

Relevant  Work  Experience 

-Average 

Perceptual  and  Motor  Skills 

-High 

How  likely  is  it  that  this  person  will  succeed?^ 
Individual  #25 


General  Intelligence 

-Low. 

Performance  in  Training 

-Low 

Relevant  Work  Experience 

-Hi  gh 

Perceptual  and  Motor  Skills 

-Hi  gh 

How  likely  is  it  that  this  person  will  succeed? 
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Individual  #  26 

General  Intelligence  -Very  Low 

Performance  In  Training  -Average 

Relevant  Work  Experience  -Average 

Perceptual  and  Motor  Skills  -Average 

How  likely  is  it  that  this  person  will  succeed? 
Individual  #27 

General  Intelligence  -Very  High 

Performance  in  Training  -High 

Relevant  Work  Experience  -Average 

Perceptual  and  Motor  Skills  -Very  Low 

How  likely  is  it  that  this  person  will  succeed?^ 
Individual  #28 


General  Intelligence 

-Very  Low 

Performance  in  Training 

-High 

Relevant  Work  Experience 

-Average 

Perceptual  and  Motor  Skills 

-Low 

How  likely  is  it  that  this  person  will  succeed?^ 
Individual  #29 

General  Intelligence  -Average 

Performance  in  Training  -Average 
Relevant  Work  Experience  -High 
Perceptual  and  Motor  Skills  -Low 

How  likely  is  it  that  this  person  will  succeed?^ 
Individual  #30 

General  Intelligence  -High 

Performance  in  Training  -low 
Relevant  Work  Experience  -Average 
Perceptual  and  Motor  Skills  -Low 

How  likely  is  it  that  this  person  will  succeed? 
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Individual  #31 


General  Intelligence  -Very  High 

Performance  In  Training  -Low 
Relevant  Work  Experience  -Average 
Perceptual  and  Motor  Skills  -Average 

How  likely  is  It  that  this  person  will  succeed? 
Individual  I  32 

General  Intelligence  -Low 

Performance  In  Training  -Average 
Relevant  Work  Experience  -Average 
Perceptual  and  Motor  Skills  -Average 

How  likely  is  it  that  this  person  will  succeed? 
Individual  #33 

General  Intelligence  -Average 

Performance  in  Training  -High 
Relevant  Work  Experience  -Very  High 

Perceptual  and  Motor  Skills  -Average 

How  likely  Is  It  that  this  person  will  succeed? 
Individual  #34 

General  Intelligence  -Low 

Performance  in  Training  -High 
Relevant  Work  Experience  -High 
Perceptual  and  Motor  Skills  -Average 

How  likely  is  it  that  this  person  will  succeed? 
Individual  #35 

General  Intelligence  -Average 

Performance  in  Training  -Average 
Relevant  Work  Experience  -Low 
Perceptual  and  Motor  Skills  -High 

How  likely  is  it  that  this  person  will  succeed? 
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Individual  f  36 

General  Intelligence  .Average 

Performance  in  Training  -Very  Low 

Relevant  Work  Experience  -Average 

Perceptual  and  Motor  Skills  -Average 

How  likely  is  It  that  this  person  will 
Individual  #37 


General  Intelligence  -Average 

Performance  in  Training  -Very  High 
Relevant  Work  Experience  -High 
Perceptual  and  Motor  Skills  -Average 
How  likely  is  it  that  this  person 
Individual  #38 

General  Intelligence  -Average 

Performance  in  Training  -Low 

Relevant  Work  Experience  -Low 

Perceptual  and  Motor  Skills  -Low 

How  likely  is  it  that  this  person 
Individual  #39 

General  Intelligence  -Average 

Performance  in  Training  -Average 

Relevant  Work  Experience  -Average 

Perceptual  and  Motor  Skills  -Very  Low 
How  likely  is  it  that  this  person 
Individual  #40 

General  Intelligence  -High 

Performance  in  Training  -Average 

Relevant  Work  Experience  -Very  Low 

Perceptual  and  Motor  Skills  -High 

How  likely  is  it  that  this  person 


will 


will 


will 


will 


succeed? 


succeed? 


succeed? 


succeed? 


succeed? 
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Individual  I  41 


General  Intelligence  -  High 

Performance  In  Training  -  Average 

Relevant  Work  Experience  -  Average 

Perceptual  and  Motor  Skills  -  High 


How  likely  is  It  that  this  person  will  succeed? 
Individual  #  42 

€eneral  Intelligence  -  Average 

Performance  in  Training  -  Average 

Relevant  Work  Experience  -  Average 

Perceptual  and  Motor  Skills  -  Average 

How  likely  is  it  that  this  person  will  succeed?^ 
Individual  #  43 


General  Intelligence 

“  Low 

Performance  in  Training 

•  Average 

Relevant  Work  Experience 

•  Average 

Perceptual  and  Motor  Skills 

-  Low 

How  likely  is  it  that  this  person  will  succeed?^ 
Individual  #  44 

General  Intelligence  -  Low 

Performance  in  Training  -  Low 

Relevant  Work  Experience  -  Low 

Perceptual  and  Motor  Skills  -Very  High 

How  likely  is  it  that  this  person  will  succeed?^ 
Individual  #  45 


General  Intelligence 

-  Average 

Performance  in  Training 

-  Average 

Relevant  Work  Experience 

-  Average 

Perceptual  and  Motor  Skills 

.  Low 

How  likely  is  it  that  this  person  will  succeed? 
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Individual  #46 

General  Intelligence  .Low 

Performance  in  Training  -Average 

Relevant  Work  Experience  -High 

Perceptual  and  Motor  Skills  -Low 

How  likely  is  It  that  this  person  will  succeed? 
Individual  #47 

6eneral  Intelligence  -Average 

Performance  in  Training  -Average 

Relevant  Work  Experience  -Average 

Perceptual  and  Motor  Skills  -Average 

How  likely  is  It  that  this  person  will  succeed? 
Individual  #48 


General  Intelligence 

-Average 

Performance  In  Training 

-Low 

Relevant  Work  Experience 

-Low 

Perceptual  and  Motor  Skills 

-Average 

How  likely  Is  it  that  this  person  will  succeed? 
Individual  #49 


General  Intelligence 

-Average 

Performance  in  Training 

-Average 

Relevant  Work  Experience 

-Low 

Perceptual  and  Motor  Skills 

-Low 

How  likely  Is  it  that  this  person  will  succeed? 
Individual  #50 


General  Intelligence 

-High 

Performance  in  Training 

-Very  Low 

Relevant  Work  Experience 

-Very  Low 

Perceptual  and  Motor  Skills 

-High 

How  likely  is  it  that  this  person  will  succeed? 
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